System and method for frame capturing and processing

ABSTRACT

Systems and methods are provided that capture and process frames of frame data. An image sensor captures frames of frame data representative of light incident upon the image sensor using a rolling shutter and outputs the frames of frame data. The image sensor captures at least one of the frames over a frame capture interval and then waits over a blanking interval before capturing another frame. A buffer receives and stores the frames output by the image sensor. An image signal processor retrieves the frames from the buffer and processes the frames over successive frame processing intervals to generate a video having a time interval per frame greater than the frame capture interval. At least one of the successive frame processing intervals is greater than the frame capture interval and is less than or equal to a sum of the frame capture interval and the blanking interval.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.16/202,408, filed Nov. 28, 2018, which is a continuation of U.S.application Ser. No. 15/444,972, filed Feb. 28, 2017, now U.S. Pat. No.10,148,882, which is a continuation of U.S. application Ser. No.15/180,529, filed Jun. 13, 2016, now U.S. Pat. No. 9,756,250, which is acontinuation of U.S. application Ser. No. 14/451,437, filed Aug. 5,2014, now U.S. Pat. No. 9,392,194, which is a continuation of U.S.application Ser. No. 14/189,967, filed Feb. 25, 2014, now U.S. Pat. No.8,830,367, which claims the benefit of U.S. Provisional Application No.61/893,805, filed on Oct. 21, 2013, each of which is incorporated hereinby reference in its entirety.

BACKGROUND

In digital camera systems, such as CMOS image sensor-based systems, thepixel array is generally read out in what is called a “rolling shutter”method. In rolling shutter, the exposure and reading of the array beginsat the top of the array for each frame and proceeds to roll down thearray throughout the frame capture interval of time. Because of this,action captured at the beginning of a frame capture interval of time iscaptured only at the top of the frame (scene), action captured in themiddle of a frame capture interval is captured in the center of theframe and action captured at the end of the frame capture interval iscaptured at the bottom of the frame. Thus, when an object moves duringthe frame capture interval (or when the camera moves during the same),the captured reproduction of the object may be a slanted or swizzledversion of the object, skewed relative to the motion between the objectand the camera over the total frame capture interval. These effects arereferred to herein as “rolling shutter artifacts”.

Rolling shutter artifacts are undesirable due to the distortion theyimpose on the object within a captured image. One method to overcomethis issue is to use a global shutter, wherein the entire image sensorarray is exposed at once, then blinded (either via mechanical shutter orelectrically-controlled light sensitivity or other). The pixel data isthen read out in a rolling manner. This approach is challenging toimplement in small camera systems and/or cameras with small image sensorpixels. Another method to minimize rolling shutter artifacts is tocapture the image frames at very high rates. While this minimizes therolling shutter time (and thus minimizes rolling shutter artifacts), ituses much more power than running systems at nominal frame rates and cancreate bottlenecks at various points within the image capturearchitecture (such as at an image sensor and/or image processor) unableto keep up with such a rapid rate of frame capture. It is thus desirableto capture images with minimized rolling shutter artifacts withoutimposing the size, weight, cost, and/or power and performancerequirements of the solutions listed above.

SUMMARY

Disclosed herein are implementations of systems and techniques for framecapturing and processing.

In an implementation, a system is provided that captures and processesframes of frame data. The system includes an image sensor that capturesframes of frame data representative of light incident upon the imagesensor using a rolling shutter and that outputs the frames of framedata, wherein the image sensor captures at least one of the frames overa frame capture interval and then waits over a blanking interval beforecapturing another frame, a buffer that receives and stores the framesoutput by the image sensor, and an image signal processor that retrievesthe frames from the buffer and processes the frames over successiveframe processing intervals to generate a video having a time intervalper frame greater than the frame capture interval, wherein at least oneof the successive frame processing intervals is greater than the framecapture interval and is less than or equal to a sum of the frame captureinterval and the blanking interval.

In an implementation, a method is provided for capturing and processingframes of frame data. The method includes capturing, by an image sensor,frames of frame data representative of light incident upon the imagesensor at a frame capture rate using a Bayer color filter array and arolling shutter, wherein at least one frame is captured over a framecapture interval before another frame is captured after a blankinginterval, storing, using a buffer, at least some of the frames capturedby the image sensor, retrieving, by an image signal processor, a subsetof the frames captured by the image sensor from the buffer according toa frame processing rate that is one half or less of the frame capturerate, and processing, by the image signal processor, the subset of theframes over successive frame processing intervals to generate a videohaving an output frame rate less than the frame capture rate.

In an implementation, a system is provided that captures and processesframes of frame data. The system includes an image sensor that capturesframes of frame data representative of light incident upon the imagesensor at a frame capture rate using a Bayer color filter array and arolling shutter and that outputs at least some of the frames of framedata, wherein the image sensor captures at least one of the frames overa frame capture interval and then waits over a blanking interval, abuffer that receives and stores at least some of the frames output bythe image sensor, and an image signal processor that retrieves, from thebuffer, a subset of the frames captured by the image sensor as retrievedframes according to a frame processing rate that is one half or less ofthe frame capture rate and that processes the retrieved frames at theframe processing rate to generate a video having an output frame rateless than the frame capture rate.

These and other aspects of the present disclosure are disclosed in thefollowing detailed description, the appended claims, and theaccompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an embodiment of a camera system,according to one embodiment.

FIG. 2a illustrates data flow through components of the camera systemwith an internal buffer, according to one embodiment.

FIG. 2b illustrates data flow through components of the camera systemwith an external buffer, according to one embodiment.

FIG. 3 illustrates data flow during image capture in a sparse capturemode, according to one embodiment.

FIG. 4 illustrates data flow during image capture in a combination mode,according to one embodiment.

FIG. 5 is a flow chart illustrating image capture in a sparse capturemode, according to one embodiment.

FIG. 6 is a flow chart illustrating image capture in a combination mode,according to one embodiment.

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

System Architecture

FIG. 1 is a block diagram illustrating an embodiment of a camera system,according to one embodiment. The camera 100 of the embodiment of FIG. 1includes an image sensor 105, a frame controller 110, an ISP 115, aninterface module 120, a storage module 125, and a processing unit 130.It should be noted that in other embodiments, the camera 100 can includedifferent or additional components than those illustrated herein. Thecomponents of the camera 100 are communicatively coupled, for instancevia a communications bus not illustrated herein.

The image sensor 105 is configured to convert light incident upon theimage sensor chip into electrical signals representative of the lightincident upon the image sensor (“image data” hereinafter). Such aprocess is referred to as “capturing” image data, and capturing imagedata representative of an image is referred to as “capturing an image”or “capturing a frame”. The image sensor can include a Bayer colorfilter array configured to capture image data and output raw Bayerpattern data. The image sensor can be configured to capture images atone or more frame rates (such as 120 frames per second, or fps, 60 fps,48 fps, 30 fps, and the like), and can be configured to capture an imagein a first interval of time and then wait a second interval of timebefore capturing another image (during which no image data is captured).The image sensor can include a charge-coupled device (“CCD”) imagesensor, a complementary metal-oxide semiconductor (“CMOS”) image sensor,or any other suitable image sensor configured to convert captured lightincident upon the image sensor chip into image data.

The frame controller 110 is configured to manipulate the capturing ofimage data by the image sensor 105 and the processing of image data bythe ISP 115, and/or to manipulate captured image data prior to theprocessing of the captured image data by the ISP. The frame controlleris discussed below in greater detail. It should be noted that althoughthe frame controller is illustrated in the embodiment of FIG. 1 is aseparate component, in practice, the frame controller can be implementedwithin another component of the camera 100, such as within the imagesensor or the ISP. In addition, in some embodiments, the image sensor isimplemented within the ISP, or the ISP is implemented within the imagesensor.

The ISP 115 is configured to process captured image data, such as imagedata captured by the image sensor 105 or image data manipulated by theframe controller 110, to produce a processed image. The processed imagecan be an image file formatted for display on a display screen, forstorage in memory, and the like. In one embodiment, the ISP performsprocessing steps including clean Bayer processing, demosaicing, and thelike, as well as compression to format the captured image data foroutput. In various embodiments, the ISP processes the image data togenerate image or video files in standard formats, such as JPEG, PNG,TIFF, AVI, or MPEG.

The interface module 120 allows a user of a camera 100 to operate thecamera, to configure the camera into one or more camera modes, tomanipulate camera settings, and the like. In one embodiment, theinterface module allows a user to configure the camera to operate in asparse capture mode (described below in greater detail), or in acombination mode (described below in greater detail). The interfacemodule can also allow a user of the camera to begin or end capturingimages or video, and can allow a user of the camera to playback or editcaptured images or video. In addition, the interface module can allow auser to output captured images or video stored within the camera (forinstance, by the storage module 125) to an external storage module,display, or the like.

The storage module 125 includes a non-transitory computer-readablestorage medium configured to store data. The storage module can includeany suitable type of storage, such as random-access memory, solid statememory, a hard disk drive, buffer memory, and the like. The storagemodule can store image data captured by the image sensor 105, forinstance prior to manipulation by the frame controller 110 or processingby the ISP 115. The storage module can also store image data forbuffering during manipulation by the frame controller, or can storemanipulated image data output by the frame controller prior toprocessing by the ISP. The storage module can store image data processedby the ISP, such as image or video files, prior to playback from thestorage module or outputting by the interface module 120 to an externalentity.

The functions described herein of the components of the camera and ofthe camera itself can be controlled by the processing unit 130. In oneembodiment, the processing unit includes a processor and a memorycomprising a non-transitory computer-readable storage medium storingcomputer-executable program instructions for capturing image data,manipulating the capture of image data, manipulating captured imagedata, processing image data, and manipulating the processing of imagedata, among other functions. In such an embodiment, the processor canexecute the computer-executable program instructions to perform thesefunctions. It should be noted that the processing unit can implementimage capture, manipulation, and processing in hardware, firmware, or acombination of hardware, firmware, and/or software. In some embodiments,the storage module 125 stores the computer-executable programinstructions for performing the functions described herein for executionby the processing unit.

FIGS. 2a and 2b illustrate data flow through components of the camerasystem with an internal buffer and an external buffer, respectively,according to one embodiment. The frame controller 110 of the embodimentsof FIGS. 2a and 2b include a sensor interface input 200, a frame writingmodule 205, a frame processor 210, a frame reading module 220, and asensor interface output 225. In addition, the frame controller of theembodiment of FIG. 2a includes a buffer 215, while the buffer isexternal to the frame controller in the embodiment of FIG. 2b . In otherembodiments, the frame controllers of FIGS. 2a and 2b include differentor additional components than those described herein.

The sensor interface input 200 receives captured image data from theimage sensor 105 and provides the received image data to the framewriting module 205. The frame writing module provides the captured imagedata to a frame processor 210 for manipulation or processing, or forstorage within the buffer 215 or within another memory not illustratedin the embodiments of FIGS. 2a and 2b . As discussed below, the frameprocessor 210 can manipulate image data stored in the buffer or receivedfrom the frame writing module. In addition, the frame processor canmanipulate the capture of image data by the image sensor and/or theprocessing of image data by the ISP 115. The frame reading module 220receives image data from the frame processor (for instance, directlyfrom the frame processor, or from the buffer or any other memory notillustrated in the embodiments of FIGS. 2a and 2b ), and provides theimage data to the sensor interface output 225, which outputs the imagedata to the ISP. In the embodiment of FIG. 2b , the buffer is locatedexternally to the frame controller, and can be located within the camera100 (for instance, within the storage module 125) or external to thecamera.

It should be noted that in some embodiments not described furtherherein, a camera 100 can implement a buffer or other memory instead of aframe controller 110. In such embodiments, an external controller (suchas the processing unit 130) can manipulate the capture of image data bythe image sensor 105 (which writes captured image data to the buffer),and can manipulate the processing of the image data stored in the bufferby the ISP 115. In addition, in some embodiments not described furtherherein, the camera 100 can implement an image capture accelerator orother companion device between the image sensor and the ISP instead of aframe controller. In some embodiments, frames are stored in the buffer215 or in the storage module 125 in a circular manner, such that framesare stored in order of capture, and new frames replace older frames whenthe buffer or storage module becomes full.

By implementing the frame controller 110, the camera 100 canbeneficially capture frames using the image sensor 105 at a higher ratethan the ISP 110 can process frames. Relatedly, by utilizing the framecontroller, a frame can be processed by the ISP over a longer timeinterval (the “frame processing interval” or “image processinginterval”) than the interval of time during which the frame is capturedby the image sensor (the “frame capture interval” or “image captureinterval”). Finally, by utilizing a frame controller, multiple framescan be captured and written to the frame controller, can be combined toproduce a combined frame, and the combined frame can be processed by theISP.

As noted above, the capture of images by the image sensor 105 while thecamera 100 is moving, while an object in the image being captured ismoving, or both, can result in rolling shutter artifacts within thecaptured image. Such rolling shutter artifacts can be reduced bymanipulating the capturing and processing of frames (in a “sparsecapture mode” as described below), or by manipulating captured frames(in a “combination mode” as described below).

Sparse Capture Mode

FIG. 3 illustrates data flow during image capture in a sparse capturemode, according to one embodiment. The image sensor 105 is configured tocapture a frame over a frame capture interval of time. In the embodimentof FIG. 3, frame capture intervals 300, 304, and 308 are shown. In someembodiments, the frame capture interval is defined by a frame capturerate, and the frame capture interval and/or frame capture rate areselected such that rolling shutter artifacts are substantially minimizedwhen capturing frames. For example, a frame capture rate of 120 fps maybe selected, resulting in a frame capture interval of 0.00833 seconds orless. Similarly, a frame capture rate of 240 fps may be selected,resulting in a frame capture interval of 0.00417 seconds or less. Theimage sensor is also configured to wait for a period of time betweenframe capture intervals without capturing frame data. These intervals oftime are referred to herein as “blanking intervals”. In the embodimentof FIG. 3, blanking intervals 302 and 306 are shown. In someembodiments, the image sensor is exposed to light during the framecapture interval, and is shielded from light during the blankinginterval.

In some embodiments, the length of the blanking interval is selectedbased on a rate that the ISP 115 can process frames. For instance, ifthe ISP can process a frame in a given length of time (a “frameprocessing interval” hereinafter), the blanking interval can be selectedsuch that the sum of the frame capture interval and the blankinginterval is greater than the frame processing interval. In someembodiments, the blanking interval is longer than the frame captureinterval, for instance by a factor of 2 or 3 or more, and in otherembodiments, the blanking interval is shorter than the frame captureinterval, for instance by a factor of 1.5, 2, or more. In someembodiments, the frame capture interval and the blanking interval areapproximately the same length. It should be noted that in theseembodiments and others, the image sensor can be configured to captureframes over a frame capture interval consistent with a first framecapture rate, but can incorporate a blanking interval between framecaptures, resulting in an effective second frame capture rate slowerthan the first frame capture rate.

The image sensor 105 captures a frame during a frame capture interval,creating frame data. The image sensor writes the frame data to a framecontroller 110. In the embodiment of FIG. 3, frame data 310 is providedby the image sensor to the frame controller after frame capture interval300, frame data 312 is provided by the image sensor to the framecontroller after frame capture interval 304, and frame data 314 isprovided by the image sensor to the frame controller after frame captureinterval 308. The frame controller outputs the frame data to the ISP115. In the embodiment of FIG. 3, frame data 316 is provided to the ISPby the frame controller at the beginning of the frame processinginterval 322, and frame data 318 is provided to the ISP by the framecontroller at the beginning of the frame processing interval 324. Insome embodiments, the image sensor sets a flag within the framecontroller indicating that the frame controller contains frame data tobe read by the ISP. The ISP can be configured to read the frame datafrom the frame controller, for instance in response to detecting framedata stored in the frame controller, in response to detecting a flag setin the frame controller indicating that the frame controller is storingframe data to be read, or in response to the ISP completing theprocessing of previous frame data.

The ISP 115 receives (or reads) frame data stored in the framecontroller associated with a captured frame, and processes the framedata during a frame processing interval. The ISP can be configured toperform any image or frame processing operations during the frameprocessing interval. In some embodiments, the ISP encodes the frame datainto a renderable or displayable image, for instance in the JPEG formator any other suitable format. In some embodiments, the frame processinginterval is substantially longer than the frame capture interval, forinstance by a factor of ×1.5, ×2, ×3, ×4, or more. The frame processinginterval can be selected based on the configuration or capabilities ofthe ISP. For example, if the ISP can process frames at a firstresolution at 60 fps, and can process frames at a second, higherresolution at 30 fps, a frame processing interval of 0.017 seconds orless can be selected when processing frame data at the first resolution,and a frame processing interval of 0.033 seconds or less can be selectedwhen processing frame data at the second resolution.

In some embodiments, the frame data captured by the image sensor 105 isvideo data. In these embodiments, the ISP 115 can process the frame dataas if the frame data were captured at a slower frame rate or over alonger frame capture interval than the image sensor captured the framedata. For instance, if the frame capture interval in which a frame iscaptured corresponds to a frame rate of 120 fps (a frame captureinterval of 0.00833 seconds followed by a blanking interval of 0.0250seconds), the ISP can process the resulting frame data such that theframe data is encoded into video over a frame processing intervalcorresponding to a frame rate 30 fps (a frame processing interval of0.0333 seconds). The ISP can output the processed frame data to a moduleor memory external to the camera 100, or can store the processed framedata in an internal memory.

The operation of the camera 100 in the sparse capture mode beneficiallyallows for 1) the capture of frames within a short enough frame captureinterval that rolling shutter artifacts are substantially minimized,while 2) capturing frames at a slow enough effective frame rate that theISP 115 can process the captured frames at a frame processing rate thatminimizes the power or resource requirements of the ISP. Although theembodiment of FIG. 3 includes a frame controller, it should be notedthat sparse capture mode as described herein can be implemented inembodiments without a frame controller. In such embodiments, the imagesensor 105 captures frame data at a frame capture interval correspondingto a frame capture rate, and outputs the captured frame data at a slowerframe rate directly to the ISP for processing at the slower frame rate.Alternatively, the image sensor 105 can capture frame data at a framecapture interval, can stop capturing frame data for a blanking intervaluntil a capture enable signal is received from the ISP, can output theframe data to the ISP for processing over a frame processing interval,and the ISP can provide a capture enable signal to the image sensor tocapture a subsequent frame.

It should be noted that in some embodiments, a sparse capture mode canbe implemented with a small or substantially zero-length blankinginterval. For example, the image sensor 105 can capture frames at aframe capture rate with minimal blanking. In these embodiments, theimage sensor can be configured to provide only a subset of capturedframes to the frame controller 110. For instance, the image sensor mayprovide only every 4th frame to the frame controller, and the ISP 115can be configured to read and process the provided frames as illustratedin FIG. 3. Alternatively, the image sensor can be configured to provideevery captured frame to the frame controller, and the ISP can beconfigured to read only a subset of the provided frames from the framecontroller. For instance, the ISP can be configured to read and processonly every 3rd frame from the frame controller, and the unread framescan be subsequently deleted or overwritten with new frames from theimage sensor.

In some embodiments, the frame controller 110 selects the length of theframe capture interval and provides the selected frame capture intervallength to the image sensor 105 for implementation. In addition, theframe controller can select the length of the blanking interval, and canprovide selected blanking interval length to the image sensor forimplementation. Likewise, the frame controller can select the length ofthe frame processing interval, and can provide the selected frameprocessing interval length to the ISP 115. The lengths of the framecapture interval, the blanking interval, and the frame processinginterval can be based on any suitable factors, such as the capabilitiesor available resources of the image sensor or the ISP, the resolution ofimage data being captured, the fill rate of a buffer or other memorywithin the frame controller, a camera setting or operation mode, or anyother suitable factor. The frame controller can also select intervallengths based on other interval lengths. For instance, the framecontroller can select the frame capture interval length and/or theblanking interval length based on the frame processing interval lengthand vice versa, can select the frame capture interval length based onthe selected blanking interval length and vice versa, and can select theframe processing interval length based on one or more of the framecapture interval length and the blanking interval length. In addition,it should be noted that a component other than the frame controller canselect one or more of the frame capture interval length, the blankinginterval length, and the frame processing interval length. For instance,these interval lengths can be selected by the image sensor, the ISP, theprocessing unit 130, or any other component or entity.

Combination Mode

FIG. 4 illustrates data flow during image capture in a combination mode,according to one embodiment. The image sensor 105 is configured tocapture a frame over a frame capture interval of time. In the embodimentof FIG. 4, several frame capture intervals 400 are illustrated. As notedabove, the frame capture interval can be defined by a frame capturerate, and the frame capture interval and/or frame capture rate areselected such that rolling shutter artifacts are substantially minimizedwhen capturing frames. The image sensor writes captured frame data 405to a buffer 215. In the embodiment of FIG. 4, frame 1, frame 2, frame 3,and frame 4 associated with the frame data 405 are written to thebuffer. The frame processor 210 combines the captured frames intocombined frames for processing. In the embodiment of FIG. 4, frame 1,frame 2, frame 3, and frame 4 are combined into a combined frame 1. Insome embodiments, the frame processor 210 and the buffer 215 areincluded within the frame controller 110, though in other embodiments,they are stand-alone components.

It should be noted that although a particular embodiment of a camera 100combination mode is illustrated in FIG. 4, in practice, a combinationmode can be implemented in a camera in a number of ways. For instance,instead of combining 4 frames, the frame processor can combine all orportions of 2 frames, 3 frames, 5 frames, or more. In some embodiments,the frame processor combines a pre-determined number of frames into acombined frame, though in other embodiments, the number of framescombined into a combined frame varies from combined frame to combinedframe, or varies based on a setting selected by a user of the camera100. Further, although the primary description of the combination modeherein focuses on the embodiment of FIG. 4, other embodiments ofoperation in a combination mode are described below.

The ISP 115 receives or reads the combined frame from the frameprocessor 210 and processes the combined frame in a frame processinginterval. For instance, the ISP receives the combined frame 1 from theframe processor and processes the combined frame 1 over the frameprocessing interval 410 to produce a processed frame. The ISP thenoutputs the processed frame to a module or memory external to the camera100, or can store the processed data in an internal memory. The ISPtemporally transcodes the combined frames into video renderable ordisplayable at a slower frame rate than the frame capture rate.

In some embodiments, the frame processor 210 requests the frames fromthe buffer 215, and in some embodiments, the buffer automaticallyoutputs the frames to the frame processor. In some embodiments, theframe processor or any other module can inform the buffer of aparticular number of frames to be combined by the frame processor, andthe buffer can indicate to the frame processor when the particularnumber of frames are stored by the buffer. In response, the frameprocessor can request the frames from the buffer, and the buffer canprovide the frames to the frame processor. Alternatively, the buffer canoutput frames to the frame processor as they as received from the imagesensor 105, and the frame processor can combine the received frames whena pre-determined number of frames are received.

The frame processor 210 can be configured to combine frames by anysuitable method. In one embodiment, the frame processor combinescorresponding pixels in each frame, for instance by digitally summingthe information of each corresponding pixel in each frame.Alternatively, the frame processor can combine corresponding pixels byaveraging the pixel information (such as the luminance or chrominanceinformation) for each of the corresponding pixels, by averaging onlysimilar pixels (corresponding pixels above a threshold measure ofsimilarity), by de-noising pixels determined to represent a stationarypart of a frame, and the like. Combining each set of correspondingpixels results in a combined frame, with each pixel of the combinedframe representing a combination of the corresponding pixels in each ofthe frames. In some embodiments, pattern noise reduction, defectivepixel correction, or any other image repair or calibration process canbe implemented by the frame processor on received frames either beforeor during combination.

The combination of frames into a combined frame prior to processing bythe ISP 115 can beneficially reduce the processing resource and powerrequirements of the ISP by reducing the total number of frames that areprocessed by the ISP. For instance, in the embodiment of FIG. 4,combining every four frames into a combined frame reduces the number offrames processed by the ISP by 75%. This beneficially allows the ISP toprocess frames at a first rate (e.g., 30 frames per second) while theimage sensor 105 captures frames at a second rate (e.g., 120 frames persecond), reducing the processing load of the ISP while increasing theamount of image information available to the ISP by increasing thenumber of captured frames (and the amount of frame data) captured by theimage sensor. By increasing the amount of frame data available to theISP for processing, the dynamic range and/or signal-to-noise ratio ofimage data processed by the ISP can be improved. Further, various imageeffects can be incorporated into the processed image data as a result ofthe increase in frame data, such as a simulated capture angle orweighted motion blur. The processed image data captured in a combinationmode can beneficially improve image quality while reducing the effect ofrolling shutter artifacts.

In one embodiment, a camera 100 can operate in a combination modewithout the use of a buffer 215. In such embodiments, the image sensor105 can directly provide captured frames to the frame processor 210. Insome embodiments, the frame processor can write combined frames to thebuffer and the ISP 115 can directly read the combined frames from thebuffer. In some embodiments, the frame processor can successivelycombine received frames as it receives the frames. In the exampleembodiment of FIG. 4, the frame processor can receive frame 1, canreceive frame 2, can add frame 1 and frame 2 to form a first summedframe, can receive frame 3, can add frame 3 to the first summed frame toform a second summed frame, and so forth. In some embodiments, framesare compressed by an image capture accelerator or other component priorto storing the frames in the buffer. In these embodiments, thecompressed frame data can be decompressed prior to being combined by theframe processor, or the frame processor can combine the compressed framedata (for instance by summing or averaging wavelet coefficients).

In some embodiments, the frame controller 110 selects the length offrame capture intervals and provides the selected frame capture intervallength to the image sensor 105 for implementation. In addition, theframe controller can select the frame capture rate, and can provide theselected frame capture rate to the image sensor for implementation. Theframe controller can also select the length of the frame processinginterval, the number of frames to combine into a combined frame, thetype of frame combination operation, or any other suitable configurationfor use in capturing and processing images. The number of frames to becombined into a combined frame can be based on properties of the framesthemselves (such as the resolution or color information of the frames),based on a selected frame capture rate by the image sensor, based on alength of frame processing interval, or based on any other suitablefactor. In addition, it should be noted that a component other than theframe controller can select one or more of the frame capture intervallength, the frame capture rate, the frame processing interval length,the number of frames to combine into a combined frame, the type ofcombination operation, and the like. For instance, any of these factorscan be selected by the image sensor, the ISP 115, the processing unit130, or any other component or entity.

Operation

FIG. 5 is a flow chart illustrating image capture in a sparse capturemode, according to one embodiment. A frame is captured 500 by an imagesensor over a first interval of time (a frame capture interval). Thecaptured frame is buffered 505, and the image sensor waits for a secondinterval of time (a blanking interval) before capturing a subsequentimage. The buffered frame is processed 515 by an ISP over a thirdinterval of time (a frame processing interval) to produce a processedframe. The frame processing interval is longer than the frame captureinterval, but is shorter than the sum of the frame capture interval andthe blanking interval.

FIG. 6 is a flow chart illustrating image capture in a combination mode,according to one embodiment. A set of frames is captured 600 at a firstframe capture rate by an image sensor. The captured set of frames isbuffered 605, and the buffered set of frames is combined 610 to producea combined frame. The combined frame is then processed 615 by the ISP ata second frame rate slower than the first frame rate to produce aprocessed frame.

Additional Considerations

Throughout this specification, some embodiments have used the expression“coupled” along with its derivatives. The term “coupled” as used hereinis not necessarily limited to two or more elements being in directphysical or electrical contact. Rather, the term “coupled” may alsoencompass two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other, or arestructured to provide a thermal conduction path between the elements.

Likewise, as used herein, the terms “comprises,” “comprising,”“includes,” “including,” “has,” “having” or any other variation thereof,are intended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus.

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Finally, as used herein any reference to “one embodiment” or “anembodiment” means that a particular element, feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for acamera expansion module as disclosed from the principles herein. Thus,while particular embodiments and applications have been illustrated anddescribed, it is to be understood that the disclosed embodiments are notlimited to the precise construction and components disclosed herein.Various modifications, changes and variations, which will be apparent tothose skilled in the art, may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope defined in the appended claims.

What is claimed is:
 1. A system that captures and processes frames offrame data, comprising: an image sensor configured to capture frames offrame data representative of light incident upon the image sensor usinga rolling shutter and at a frame capture rate; a buffer configured tostore combined frames from a frame processor at a frame rate less thanthe frame capture rate, wherein the frame processor combines at leasttwo of the frames; and an image signal processor configured to processthe combined frames at less than the frame capture rate, wherein anumber of frames that are combined is selectable using an input to theimage signal processor.
 2. The system of claim 1, wherein the number offrames is configurable from one combined frame to at least anothercombined frame.
 3. The system of claim 1, wherein the frame processor isconfigured to combine the frames by summing or averaging waveletcoefficients associated with the captured frames.
 4. The system of claim1, wherein the combined frames incorporate one or more image effects. 5.The system of claim 1, wherein the combined frames are produced with agreater dynamic range than the captured frames.
 6. The system of claim1, wherein the combined frames are produced using a weighted motionblur.
 7. The system of claim 1, wherein the combined frames are producedusing a simulated capture angle.
 8. The system of claim 1, wherein theframe processor produces the combined frames by successively combiningframes retrieved from the buffer.
 9. A method for capturing andprocessing frames of frame data, comprising: capturing frames of framedata representative of light incident upon an image sensor using arolling shutter at a frame rate; storing combined frames, produced by aframe processor using some of the frames, at another frame rate that isless than the frame rate; and processing the combined frames at theanother frame rate using an image signal processor, wherein a number offrames that are combined is selectable using an input to the imagesignal processor.
 10. The method of claim 9, wherein the number offrames that are combined varies per combined frame.
 11. The method ofclaim 9, wherein the combined frames are produced by summing oraveraging wavelet coefficients associated with the captured frames. 12.The method of claim 9, wherein the combined frames are produced with agreater dynamic range than the captured frames.
 13. The method of claim9, wherein each combined frame includes at least one image effect. 14.The method of claim 13, wherein the at least one image effect is aweighted motion blur.
 15. The method of claim 13, wherein the at leastone image effect is a simulated capture angle.
 16. The method of claim9, wherein the frame processor produces the combined frames bysuccessively combining frames retrieved from the buffer.
 17. Anapparatus, comprising: an image sensor configured to capture frames offrame data representative of light incident upon the image sensor at afirst frame rate, wherein exposure and readout from the image sensorscans across the image sensor; a buffer configured to store combinedframes, the combined frames produced by a frame processor using some ofthe frames, at a second frame rate less than the first frame rate; andan image signal processor configured to process the combined frames atthe second frame rate, wherein a number of frames that are combined isselectable using an input to the image signal processor.
 18. Theapparatus of claim 17, wherein each combined frame includes at least oneof a simulated capture angle and a weighted motion blur.
 19. Theapparatus of claim 18, wherein the frame processor is configured tocombine the frames by summing or averaging wavelet coefficientsassociated with the captured frames.
 20. The apparatus of claim 17,wherein the combined frames incorporate one or more image effects.